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Preface 


This  document  contains  a summary  description  of  the 
philosophy  and  technical  approach  underlying  the  evaluation 
program  that  TSC  conducts  for  UMTA's  Office  of  Service  and 
Methods  Demonstrations.  Much  of  the  material  included 
herein  was  initially  prepared  for  the  Service  and  Methods 
Demonstration  Program  FY76  Annual  Report . This  document  has 
been  published  and  distributed  separately  for  review  and 
comment  by  a wide  range  of  analysts  and  planners  with 
expertise  in  areas  and  issues  related  to  evaluation.  The 
contents  are  intended  to  serve  as  the  basis  for  forthcoming 
versions  of  the  report.  Evaluation  Guidelines  for  Service 
and  Methods  Demonstrations.  Until  a revised  version  of  the 
Guidelines  is  completed,  this  document,  along  with  the 
existing  version  of  the  Guidelines  and  other  written 
material  furnished  by  TSC,  will  provide  direction  for 
conducting  Service  and  Methods  Demonstration  project 
evaluat ions . 
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1. 


INTRODUCTION 


UMTA's  Service  and  Methods  Demonstration  (SMD)  Program 
has  the  objective  of  improving  existing  transit  operations 
by  sponsoring  the  development  and  implementation  of  new 
techniques  and  services  on  a nationwide  basis.  These 
innovations,  which  rely  on  existing  transit  technology,  are 
intended  to  produce  short  range  improvements  in  the  quality 
and  quantity  of  public  transportation.  Frequently  this 
requires  a combination  of  modal  improvements,  integrated  and 
coordinated  in  order  to  supply  a variety  of  transportation 
services  for  various  users,  trip  purposes,  and  travel 
patterns.  The  program  also  aims  to  reduce  institutional  and 
regulatory  friction  that  so  often  blocks  innovation. 

Further  it  seeks  to  achieve  a more  efficient  balance  of 
travel  demand  among  modes,  times  of  day,  and  geographic 
areas.  The  use  of  physical  and  operational  techniques  to 
increase  the  capacity  of  existing  facilities  is  also  being 
demonstrated  and  evaluated  in  the  program. 

The  UMTA  Service  and  Methods  Demonstration  Program 
pursues  demonstration  projects  and  studies  in  four  major 
program  areas:  traffic  management,  paratransit,  service  for 

transit  dependents,  and  price  and  service  improvements. 

Traf f ic  Management  emphasizes  priority  treatment  for 
transit  and  other  high  occupancy  vehicles  to  expedite  peak 
period  movement  of  passengers.  Types  of  projects  include 
exclusive  busways,  reserved  lanes  on  freeways,  arterials, 
and  city  streets;  signal  preemption,  transit  malls,  and  auto 
restricted  zones. 

Paratransit  includes  a broad  range  of  services  that 
occupy  the  transportation  spectrum  between  conventional 
transit  and  the  private  auto,  i.e. , dial-a-ride,  jitney, 
vanpools,  taxis,  subscription  buses,  and  other  forms  of  ride 
sharing.  The  main  intent  is  to  provide  improved  service  and 
to  increase  vehicle  occupancy. 

Service  for  Transit  Dependents  seeks  to  develop 
specialized  services  that  will  address  the  needs  of  the 
transit  dependent  person-- the  elderly,  handicapped,  young, 
and  poor.  Included  are  novel  methods  to  improve  inner  city 
circulation,  "reverse'1  commuting,  specialized  equipment  for 
elderly  and  handicapped  transportation,  subscription 
services,  demand-responsive  services,  and  user  side 
subsidies. 

Price  and  Service  Improvements  seek  to  increase  transit 
patronage  and  reduce  auto  usage  through  the  exploration  of  a 
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variety  of  price-related  (i.e.,  economic)  incentives,  disin- 
centives, and  service  improvements. 

The  SMD  Program  has  established  five  major  objectives 
listed  below  which  would  be  clearly  indicative  of  improved 
transportation  services  and  which  could  be  attained  through 
conduct  of  the  demonstrations: 

Reduce  travel  time  by  transit 

Increase  the  area  coverage  of  transit  service 

Improve  the  reliability  of  transit  service 

Increase  the  productivity  of  transit  vehicles 

Improve  the  mobility  of  transit  dependents. 

These  objectives  are  subject  to  modification  or  augmentation 
as  national  and  local  priorities  and  needs  change. 

A crucial  aspect  of  the  Service  and  Methods 
Demonstration  Program  is  the  performance  of  technically 
sound  and  objective  evaluations  of  the  individual 
demonstration  projects.  Under  UMTA  sponsorship,  the 
Transportation  Systems  Center  (TSC)  of  the  U. S.  Department 
of  Transportation  conducts  a broad  program  of  demonstration 
evaluation,  evaluation  methodology  development,  and  research 
in  support  of  these  activities.  At  present,  TSC  has  full 
technical  and  programmatic  responsibility  for  a program  of 
demonstration  evaluation.  In  order  to  obtain  technically 
sound,  objective,  and  consistent  evaluations  of 
demonstration  projects  which  will  increase  the  understanding 
and  transferability  of  their  operations  and  impacts,  TSC  has 
established  a single,  coordinated  demonstration  evaluation 
program. 

The  various  demonstrations  are  intended  to  serve  as 
either  real-world  experiments  involving  innovative  service 
concepts  or  techniques  implemented,  or  as  exemplary  models 
to  be  applied  or  adapted  by  other  locales  across  the 
country.  Accordingly,  the  focus  of  the  evaluations  is 
threefold:  (1)  to  assess  the  institutional  and  operational 

feasibility  of  the  demonstration  concepts  and  techniques; 

(2)  to  assess  the  transportation  and  socioeconomic  impacts 
of  the  demonstration  project;  and  (3)  to  provide  guidance, 
based  on  operational  experience  during  the  demonstration, 
for  future  applications  of  the  concepts  and  techniques. 

These  evaluations  deal  with  actual  events  and  impacts  and 
should  be  differentiated  from  before-the-fact  evaluation  of 
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potential  impacts  more  commonly  encountered  in  the 
comparison  of  service  alternatives  performed  in 
transportation  planning  studies.  In  addition  to  their 
specific  utility  to  the  SMD  Program,  the  demonstration 
evaluations  also  provide  increased  knowledge  essential  for 
improved  urban  transportation  planning  and  national  policy 
formulation. 

This  paper  describes  the  philosophy  and  approach 
underlying  SMD  evaluations.  The  contents  include  a 
discussion  of  the  following:  (1)  an  outline  of  the  SMD 

evaluation  philosophy;  (2)  a description  of  the  evaluation 
process  and  current  methodology  development;  (3)  an  analytic 
framework  for  conducting  SMD  evaluations;  and  (4)  a detailed 
description  of  specific  elements  within  the  SMD  evaluation 
f ramework. 


2.  EVALUATION  PHILOSOPHY 

The  SMD  evaluations  are  structured  around  three  basic 
questions : 

What  changes  were  made  to  the  transportation  system? 

What  were  the  impacts  of  these  transportation  changes? 

Why  did  these  impacts  occur? 

As  will  be  explained  in  the  following  sections,  these 
questions  are  addressed  through  careful  documentation  of  the 
events  and  circumstances  surrounding  the  implementation  and 
operation  of  the  project,  as  well  as  a detailed  analysis  of 
impacts  and  cause  and  effect  relationships.  Demonstration 
evaluations  are  not  designed  to  judge  the  capability  or 
performance  of  the  grantee,  nor  do  they  emphasize 
classifying  demonstrations  as  successes  or  failures. 

Rather,  an  important  premise  of  the  SMD  Program  is  that 
every  demonstration  is  of  value  if  it  increases  knowledge 
about  innovative  transit  service  concepts  and  techniques  and 
fosters  innovation  in  other  locales. 

The  SMD  Program  attempts  to  maximize  the  quality  and 
utility  of  information  gained  from  the  demonstrations  by 
developing  and  employing  a consistent,  carefully  structured 
approach  to  demonstration  evaluation.  Each  evaluation  is 
built  around  the  basic  analytical  framework  described  in  the 
next  section,  with  emphasis  placed  on  using  state-of-the-art 
data  collection  and  analysis  techniques  which  are  consistent 
from  the  standpoint  of  efficiency,  accuracy,  and  output. 
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This  stress  on  consistency  does  not,  however,  preclude 
recognition  of  the  unique  learning  potential  within  each 
demonstration.  since  demonstrations  vary  in  terms  of 
objectives,  relevant  issues,  complexity  content,  and 
context,  the  scope  and  emphasis  of  each  evaluation  must  be 
tailored  to  the  specific  characteristics  of  the 
demonstration. 

In  view  of  the  nature  and  relatively  short  (two  to 
three  year)  time  frame  of  SMD  projects,  the  evaluations 
typically  emphasize  examination  of  short-run  impacts.  These 
primarily  include  impacts  on  users,  transportation 
operators,  and  other  groups  which  occur  during  the 
demonstration  period.  Only  under  special  circumstances 
(e.g.  , a major  transportation  change  in  a large  area  or 
corridor)  is  an  analysis  conducted  of  longer-run  changes 
such  as  land  use  changes  (e.g.,  employment  and  residential 
location)  . 

Whatever  the  mix  of  short  and  long-run  impacts 
examined,  strong  attempts  are  made  in  the  design  and  conduct 
of  demonstration  evaluations  to  monitor  the  presence  of,  and 
isolate  the  effects  of,  exogenous  (i.e.,  non-demonstration- 
related)  factors  such  as  an  economic  recession,  fuel 
shortage,  or  change  to  other  parts  of  the  transportation 
system.  Demonstration  evaluation  planning  generally  takes 
place  well  in  advance  of  demonstration  implementation  to 
permit  measurement  of  key  variables  of  interest  before, 
during,  and  after  the  project  begins.  In  addition  to  the 
application  of  before-after  experimental  designs, 
evaluations  can  involve  the  monitoring  of  a control  group  or 
area,  as  well  as  the  test  group  or  area.1 * * *  For  example,  the 
analysis  of  travelers5  behavioral  response  to  a new  or 
improved  transportation  service  typically  includes  non-users 
as  well  as  users. 


lA  comprehensive  discussion  of  various  experimental  design 

approaches  can  be  found  in  Charles  River  Associates, 

Measurement  of  the  Effects  of  Transportation  Changes, 
September  1972,  Chapter  4,  and  in  Donald  T.  Campbell  and 

Julian  C.  Stanley,  Experimental  and  Quas i-Exper imental 

Designs  for  Research. 
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3. 


EVALUATION  PROCESS  AND  METHODOLOGY  DEVELOPMENT 


As  stated  earlier,  TSC  has  responsibility  for 
coordinating  the  evaluation  planning  and  implementation 
process  to  foster  consistency  in  the  output  of  individual 
project  evaluations.  TSC's  functions  include:  (1) 

establishing  standardized  evaluation  procedures  to  be 
applied  in  all  demonstration  evaluations;  (2)  specifying  the 
desired  output  and  scope  of  individual  evaluations  and 
providing  close  technical  supervision  of  a team  of 
contractors  who  perform  the  evaluations;  (3)  developing 
improved  methodology  for  demonstration  evaluation;  and  (4) 
devising  and  implementing  strategies  for  cross-cutting 
analysis  of  demonstrations. 

The  report  Evaluation  Guidelines  for  Service  and 
Methods  Demonstration  Proiects2  represents  initial  efforts 
on  the  part  of  TSC  to  structure  the  evaluation  process. 

This  document  describes  the  basic  time-sequenced  process  to 
be  followed  in  planning  and  conducting  an  evaluation  and  the 
nature  of  the  liaisons  among  the  various  organizations 
involved  in  the  demonstration  (UMTA,  TSC,  evaluation 
contractor,  grantee,  other  local  organizations).  It  also 
contains  definitions  of  measures  to  be  used  for  level-of- 
service  and  impact  assessment  (with  major  concentration  on 
the  five  SMD  Program  objectives) , recommended  data 
collection  and  analysis  procedures  for  specific  measures, 
and  guidelines  regarding  survey  and  statistical  methodology. 
This  document  is  intended  to  undergo  periodic  updating  to 
reflect  actual  evaluation  experience  and  refinements  in 
evaluation  philosophy  and  techniques. 

The  demonstration  evaluation  process  has  two  major 
components,  evaluation  planning  and  evaluation  performance. 
In  general,  the  planning  phase  begins  with  the  preparation 
(usually  by  TSC)  of  an  Evaluation  Framework  which  describes 

(1)  pertinent  information  on  the  project  and  its  settings: 

(2)  SMD  Program  and  relevant  national  and/or  local 
objectives  addressed;  (3)  key  questions  or  issues  to  be 
examined;  and  (4)  recommended  scope,  focus,  and  approach  for 
the  evaluation.  The  Evaluation  Framework  then  serves  as  the 
basis  (along  with  the  general  Evaluation  Guidelines)  for  the 


2Heaton,  Carla,  Chester  McCall  and  Robert  Waksman. 
Evaluation  Guidelines  for  Service  and  Methods 
Demonstration  Projects,  Report  No.  UMTA-MA-06-0049- 
76-16,  February,  1976. 
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development  of  an  Evaluation  Plan  (generally  prepared  by  an 
evaluation  contractor) . The  latter  document  specifies  in 
detail  the  proposed  evaluation  design  and  analysis 
framework,  data  requirements,  data  collection  methodology, 
analysis  techniques,  and  technical  management  plan  and 
resources  necessary  to  evaluate  the  demonstration' s impacts 
and  its  potential  applicability  to  other  locales. 

The  active  phase  of  the  evaluation  involves  collection 
and  analysis  of  data  relative  to  transportation  and 
socioeconomic  impacts  and  preparation  of  various  types  of 
evaluation  reports.  Data  collection  is  usually  performed  by 
the  demonstration  grant  recipient  as  part  of  the 
demonstration.  TSC  and/or  an  evaluation  contractor  is 
responsible  for  providing  technical  guidance  to  the  grantee 
regarding  data  collection  requirements  and  methodology  as 
well  as  for  monitoring  all  of  the  data  collection  activities 
carried  out  by  the  grant  recipient.  Analysis  of  the  data, 
synthesis  of  findings  for  transferability,  and  preparation 
of  various  evaluation  reports  is  the  responsibility  of  TSC 
and  the  evaluation  contractors.  These  reports  and  the 
results  contained  therein  serve  as  the  basis  for  the  Service 
and  Methods  Demonstration  Annual  Report  and  are  used  in  a 
variety  of  cross-cutting  analyses.  Techniques  are  currently 
being  developed  and  applied  for  comparing  demonstration 
results  across  sites,  both  within  and  across  demonstration 
service  concepts.  The  results  obtained  should  serve  to 
enhance  the  transferability  of  the  demonstration  concepts  by 
leading  to  an  understanding  of  what  factors  have  been  most 
influential  on  project  outcome  and  indicating  how  the 
results  would  differ  under  other  circumstances. 

Owing  to  the  technical  difficulty  of  performing 
evaluations  of  projects  which  take  place  in  real-world, 
dynamic  settings  and  the  absence  of  a ready-to-use 
conceptual  framework  or  methodology  for  evaluation,  the  SMD 
Program  provides  a unique  and  challenging  opportunity  to 
develop  and  apply  innovative  evaluation  techniques. 
Considerable  emphasis  is  placed  on  employing  the  most  up-to- 
date  evaluation  methodology  so  as  to  enhance  the  efficiency 
and  accuracy  of  the  evaluation  process.  Moreover,  there  is 
emphasis  on  developing  and  testing  novel  data  collection  and 
analysis  techniques  which  have  potentially  broad  application 
within  the  program.  This  methodology  development  evolves 
through  formal  analytical  studies,  which  may  have  a broader 
focus  than  demonstration  evaluation,  and  through  informal 
efforts  spurred  by  deficiencies  in  existing  methodology.  As 
the  results  of  these  efforts  become  available,  an 
appropriate  approach  will  be  selected,  incorporated  into  the 
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Evaluation  Guidelines,  and  then  applied  to  ongoing  and 
future  SMD  projects. 


4.  ANALYTICAL  FRAMEWORK  FOR  SMD  EVALUATIONS 

The  body  of  theory  and  analytical  techniques  which  has 
evolved  for  urban  transportation  systems  planning  and 
analysis  has  largely  been  concerned  with  before-the-fact 
comparison  of  prospective  transportation  and  socioeconomic 
impacts  of  alternative  transportation  systems.3  A conceptual 
framework  and  format  for  evaluation  has  been  developed  which 
is  specifically  appropriate  for  performing  evaluations  of 
transit  demonstration  projects.  It  is  designed  to  permit 
determination  of  cause  and  effect  relationships  and 
contributions  of  innovative  elements  of  an  operating  transit 
system.  This  framework,  based  on  principles  of 
transportation  supply  and  demand  analysis,  not  only  enhances 
the  consistency  and  comparability  of  evaluations,  but  also 
permits  a comprehensive  assessment  of  project  feasibility 
and  impacts.  Moreover,  as  will  be  seen,  this  framework  is 
consistent  in  emphasis  with  the  SMD  Program  objectives 
involving  travel  time,  coverage,  reliability,  productivity, 
and  transit  dependent  service. 

Figure  1 depicts  the  basic  supply-demand  framework  of 
transportation  change  which  should  be  applied  to  SMD 
evaluations.  The  demonstration  elements,  no  matter  how  many 
or  how  complex,  serve  to  alter  the  characteristics  of 
transportation  supply,  i.e.,  the  number  of  travel  options 
available  and/or  the  level  of  service  attributes  of  those 
options  (e.g.,  travel  time,  reliability,  convenience)  as 
viewed  by  potential  users. 

In  response  to  these  supply  changes,  individuals  make 
short-run  decisions  regarding  travel  frequency  (and  whether 
to  travel  at  all) , spatial  patterns  of  travel,  mode,  and 
time-of-day  on  the  basis  of  personal  preferences  for  these 
attributes.  The  aggregate  effect  of  individual  behavioral 
responses  to  supply  changes  is  manifested  in  the  level  of 
demand  for  each  service  mode  affected  by  the  demonstration. 
The  interaction  of  supply  changes  and  demand  responses 
produces  levels  of  supply  and  demand  which  differ  from  the 


3 See  for  example.  Measurement  of  the  Effects  of  Transportation 
Changes , op.  cit. , Chapter  3 and  Marvin  Manheim, 

Fundamentals  of  Transportation  Systems  Analysis,  1974. 
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FIGURE  1.  SUPPLY /DEMAND  FRAMEWORK  FOR 
DEMONSTRATION  EVALUATION 
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pre-demonstration  levels.  The  ultimate  effect  of  the 
demonstration  is  measured  in  terms  of  impacts  on  users, 
suppliers,  and  other  groups  as  appropriate. 

In  addition  to  tracing  the  effects  of  a supply-side 
change  on  demand,  the  conceptual  framework  for  SMD 
evaluations  also  recognizes  that  because  of  the 
interdependence  of  supply  and  demand,  changes  in  demand  may 
affect  level  of  service.  For  instance,  in  demand-responsive 
systems  travel  time  and  reliability  are  a function  of  levels 
of  demand. 


5.  CONTENT  OF  DEMONSTRATION  EVALUATIONS 

In  order  to  enhance  the  consistency  of  the  SMD 
evaluations,  a standardized  approach  for  evaluations  has 
been  developed.  This  standardized  approach  involves  a 
detailed  description  of  the  demonstration  background, 
objectives,  setting,  implementation  process,  and  operations 
as  well  as  an  analysis  of  supply  and  demand  changes  and  the 
impact  of  these  changes  on  users,  suppliers,  and  other 
groups  of  interest.  The  components  of  demonstration 
evaluations  are  discussed  below,  in  the  order  in  which  they 
should  appear  in  a final  evaluation  report. 


5 . 1 Demonstration  Background  and  Objectives 

Demonstration  evaluation  reports  begin  with  a 
description  of  the  significant  factors  and  events  leading  up 
to  the  grant  application,  for  instance,  prior  transit 
innovations  or  planning  studies  which  underscored  the  need 
for  the  demonstration  service  concept.  In  addition,  there 
is  a discussion  of  the  various  local  and  national  objectives 
and  issues  addressed  by  the  project,  which  in  turn  form  the 
basis  for  the  evaluation.  It  should  be  noted  that  local 
objectives  or  rationale  for  the  demonstration  do  not  always 
coincide  with  the  SMD  Program  objectives  or  with  the  major 
issues  of  national  importance,  in  that  they  may  be  related 
to  specific  problems  within  the  site  or  a specific 
performance  criterion.  The  evaluation  must  be  tailored  to 
produce  generalizable  knowledge  from  the  local  experience. 


5.2  Demonstration  Setting 

An  understanding  of  the  demonstration  site  is  crucial 
not  only  for  the  purpose  of  understanding  the  outcome  of  the 
demonstration  but  also  for  enhancing  the  transferability  of 
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the  demonstration*  The  demographic  composition  of  the 
service  area  — for  example,  population  density,  employment 
density,  age  composition,  auto  owner ship- -may  be  an 
important  determinant  of  the  acceptance  and  use  of  the 
demonstration  service.  Moreover,  it  is  necessary  to  predict 
the  applicability  of  the  concept  for  other  potential  sites. 
Similarly,  the  transportation  characteristics  of  the  area 
prior  to  the  demonstation--i.e. , supply  of  transportation  by 
mode  and  provider,  travel  patterns,  institutional  factors — 
must  be  understood  to  provide  a basis  for  comparison  with 
the  situation  after  the  demonstration  is  implemented  and  to 
furnish  insight  into  the  demonstration  outcome.  Most 
important,  there  must  be  continual  awareness  throughout  the 
evaluation  process  of  exogenous  changes  or  other  site- 
specific  factors  relevant  to  demonstration  implementation, 
operations,  or  outcome. 


5. 3 Demonstration  I mp le me nt a ti on  and  Operations 

A lucid  documentation  of  the  planning/implementation 
process  and  operations  is  needed  to  understand  the 
viability,  impacts,  and  transferability  of  the  demonstration 
concept.  This  description  may  include  preparatory  steps 
such  as  personnel  training  and  regulatory  changes  as  well  as 
the  overall  phasing  and  management  of  the  project.  Although 
reflecting  site-specific  conditions,  the  experience  of  the 
demonstration  site  in  implementing  the  service  can  provide 
generally  applicable  guidance  to  other  locales  on  possible 
roadblocks  to  implementation,  steps  required  to  overcome 
these  obstacles,  and  a representative  time  period  and 
resource  level  to  allow  for  accomplishing  these  steps.  The 
description  of  demonstration  operations  covers  the  services, 
equipment,  management  techniques,  fares,  marketing,  and 
other  innovative  aspects  of  the  demonstration.  Where 
relevant,  there  is  a discussion  of  problems  encountered  and 
solutions  adopted,  which  again  can  be  useful  to  other  sites. 
The  SMD  projects,  as  documented  in  evaluation  reports,  thus 
provide  an  opportunity  for  improving  the  state-of-the-art  of 
transportation  planning  and  transportation  systems 
management . 

There  is  recognition  of  the  fact  that  no  two  locales 
will  "activate”  a particular  concept  or  technique  in  the 
same  manner.  In  other  words,  differences  in  implementation 
and  operational  procedures  will  result  in  a range  of  concept 
applications.  Because  of  these  subtle  differences,  and  also 
because  of  the  fact  that  it  is  useful  to  plan  variations  of 
a given  concept  or  technique,  the  SMD  Program  sometimes 
funds  several  applications  in  order  to  ascertain  the 
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viability  and  impacts  of  the  concept  or  technique  under 
different  circumstances.  However,  even  when  there  is  only 
one  demonstration  funded  in  a particular  category,  a key 
aspect  of  the  demonstration  evaluation  is  to  carefully 
document  the  factors  specific  to  that  application--namely , 
project  background,  site  characteristics,  implementation 
process,  and  operations. 


5.4  Level -of -Service  (Supply)  Changes 

The  analysis  of  level-of- service  changes  involves 
portraying  in  specific,  quantitative  terms,  the  effects  of 
the  demonstration  project  on  the  urban  transportation  system 
as  perceived  by  potential  users.  Depending  on  the  nature  of 
the  demonstration,  one  or  more  of  the  following 
transportation  supply  attributes  can  be  affected:  the 

choice  of  available  modes  or  submodes  and  the  level-of- 
service  attributes  of  these  choices  such  as  coverage,  travel 
time,  reliability,  fare,  comfort,  and  other  amenities. 
Current  SMD  projects  tend  to  emphasize  service  innovations 
or  techniques  which  are  expected  to  improve  coverage,  travel 
time,  and  reliability — three  of  the  five  Program  objectives. 

Coverage  is  a service  attribute  which  refers  to  the 
accessibility  of  travelers  to  the  system.  Spatial  coverage 
improvements  can  occur  as  a result  of  transit  service 
expansion  or  new  service  within  an  area  not  previously 
served  by  public  transportation  or  by  the  replacement  of  a 
fixed-route  service  with  a door-to-door  demand- responsive 
service.  Temporal  coverage  improvements  can  be  achieved 
through  expanded  time  periods  of  service  operation  or 
increased  frequency  of  operation  within  the  same  time 
period.  Because  of  the  somewhat  innovative  and  complex 
methods  of  increasing  transit  coverage  applied  in  the  SMD 
Program  (i.e.,  emphasis  on  submodes  such  as  paratransit 
services  rather  than  conventional  fixed-route  bus  service) , 
the  analysis  of  coverage  changes  has  required  the 
identification  of  new  types  of  quantitative  measures  to 
express  coverage  changes. 

Travel  time  improvements  can  be  made  through  traffic 
management  techniques  such  as  reserved  bus  lanes  and 
operational  changes  involving  equipment,  schedules,  or 
dispatching.  Since  it  is  known  that  users  perceive 
different  elements  of  a door-to-door  trip  differently,  the 
analysis  of  travel  time  changes  entails  segmentation  of  time 
savings  into  components  such  as  access  time,  wait  time,  ride 
time,  and  transfer  time. 


Service  reliability  improvements  may  come  about  from 
some  of  the  same  techniques  as  result  in  travel  time 
savings,  as  well  as  from  specific  innovations  designed  to 
reduce  the  variability  of  one  or  more  travel  time 
components.  The  analysis  of  reliability  changes  has  proved 
to  be  highly  complex  and  difficult.  This  is  due  to  the 
paucity  of  identifiable  measures  to  quantify  such  changes, 
the  difficulty  of  ascertaining  which  aspects  of  service 
affect  reliability,  and  the  fact  that  the  concept  of 
reliability  varies  across  modes.  In  response  to  this 
problem,  an  analytical  study  is  underway  to  identify  an 
appropriate  conceptual/analytical  framework  for  dealing  with 
reliability  changes. 

Over  and  above  ascertaining  the  effect  of  a 
demonstration  on  the  three  SMD  objectives,  as  relevant,  the 
evaluations  of  SMD  projects  also  examine  qualitative  level- 
of- service  attributes  such  as  safety,  convenience,  and 
vehicle  amenities.  Since  a demonstration  may  improve  some 
service  attributes  at  the  expense  of  others,  the  analysis  of 
level-of- service  changes  attempts  to  understand  the  various 
trade-offs  involved.  Moreover,  segmentation  of  measures  by 
socioeconomic  characteristics,  time-of -day , trip  purpose,  or 
other  means  is  generally  performed  in  order  to  understand 
the  differential  benefits  of  level-cf- service  changes  to 
different  population  groups. 


5.5  Travel  Behavior  (Demand)  Changes 

In  response  to  changes  in  the  level  of  service  provided 
by  the  transportation  system,  individuals  within  the  service 
area  are  apt  to  alter  their  travel  choices  in  some  fashion-- 
e.g.,  make  a given  trip  by  a different  travel  mode  or  in  a 
different  time  period,  make  a given  type  of  trip  to  a 
different  destination,  make  additional  trips.  The  analysis 
of  travel  behavior  changes  involves  measurement  at  the 
individual  level  and  at  the  system  level. 

In  terms  of  system  level  effects,  it  is  useful  to 
collect  and  analyze  data  continually  before  and  throughout 
the  demonstration  on  ridership,  disaggregated  by  time-of- 
day,  day-of-week,  trip  purpose,  service  type,  and  market 
group,  as  available.  Moreover,  aggregate  statistics  on 
market  penetration  (percent  of  eligible  persons  using 
service,  with  eligibility  defined  in  terms  of  certain 
criteria) , mode  split  (percent  of  trips  by  each  mode  or 
submode)  , and  spatial  patterns  of  travel  (origin-destination 
volumes)  collected  at  several  points  before  and  during  the 
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project,  are  useful  indicators  of  the  demonstration's  effect 
on  overall  volume  and  flow  of  travel. 

At  the  individual  level,  the  analysis  of  travel 
behavior  examines  changes  in  trip  length,  destination 
choice,  and  trip  frequency,  all  stratified  by  trip  purpose. 
In  general,  travel  and  socioeconomic  characteristics  and 
attributes  of  non-users  as  well  as  users  are  analyzed  in 
order  to  isolate  the  effects  of  the  transportation  and  to 
gain  insight  into  market  penetration  and  usage  statistics. 
Analysis  of  demand  elasticities  is  also  performed  to 
understand  the  user's  sensitivity  to  individual  level-of- 
service  attributes  within  or  across  modes. 


5 . 6 Operator  Impacts  and  Productivity 

Changes  in  level  of  service  and  demand  will  have  a 
combined  effect  on  an  operator's  ability  to  serve  the  public 
in  a satisfactory,  yet  economical  manner.  In  providing 
service,  the  operator  must  be  conscious  of  operating  costs 
incurred  and  the  effective  use  of  available  resources. 

Analysis  of  operator  impacts  focuses  on  (1)  the  net 
costs  of  operations,  (2)  the  utilization  of  vehicle 
capacity,  and  (3)  the  cost-effectiveness  of  transit 
services. 

Cost  analysis  considers  cost  and  revenue  elements 
stratified  by  service  type  and  by  other  factors  (e.g. , 
administrative,  operating) , where  appropriate.  Attention  is 
also  given  to  cost/revenue  ratios  and  the  impacts  of 
operating  strategies  upon  cost  components. 

Improvements  in  the  utilization  of  vehicles  can  stem 
from  better  allocation  of  vehicles  or  from  increased 
patronage.  Analysis  measures  include  a demand  (user) 
element  and  a vehicle  allocation  element  (e.g., 
passengers/vehicle  hour) . This  set  of  measures  is  often 
referred  to  as  "vehicle  productivities,"  and  improvement  of 
transit  vehicle  productivity  is  one  of  the  five  Service  and 
Methods  Demonstration  objectives.  Vehicle  productivities 
are  segmented  by  service  type,  and  also  by  vehicle  type  in 
cases  where  more  than  one  vehicle  is  used  to  provide  a 
particular  service. 

Improvements  in  the  cost-effectiveness  of  providing 
service  are  frequently  an  important  impact  of 
demonstrations.  Measures  include  a cost  element  and  an 
operator  element  (e.g.,  cost/vehicle  hour),  usually 
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segmented  by  service  type  and  time  of  day.  In  many  cases, 
examination  of  cost-effectiveness  data  can  indicate  that  cost 
savings  can  be  achieved  at  a given  service  level  through 
modification  of  service  policies  (e. g. r by  substituting  a 
different  vehicle  during  certain  periods  of  the  day)  . 

Analysis  of  productivities  and  economics  is  not  complete 
without  consideration  of  factors  which  can  explain  resulting 
levels  of  productivities  and  efficiencies.  Issues  which  are 
typically  considered  include  management  policies,  driver  work 
rules,  vehicle  failures,  and  the  size  of  the  service  area. 


5 .7  Non-Travel  Impacts 

Analysis  of  travel  behavior,  productivities,  and  efficiency 
address  direct  impacts  of  the  demonstration  on  users  and 
operators.  There  are,  however,  broader  impacts  of  the 
demonstration  on  these  groups  as  well  as  extended  effects  on 
groups  not  directly  involved  in  the  demonstration  services. 

These  effects  are  often  gualitative  in  nature,  but  should  also  be 
included  in  the  analysis. 

Transit  dependent  groups,  particularly  elderly  and 
handicapped  people,  experience  improvements  in  mobility, 
which  in  turn  may  imply  increased  participation  in  a wider 
range  of  social  activities,  increased  access  to  medical  care,  and 
an  improved  sense  of  well-being  and  independence  from  others. 
Similarly,  the  young  and  the  poor  who  lack  access  to  an 
automobile  may  experience  greater  educational/ 

recreational/employment  opportunities  coupled  with  a reduction 
in  the  cost  of  travel  and  less  dependence  on  friends  and 
relatives.  Since  improved  service  to  the  transit  dependent  is 
one  of  the  five  SMD  objectives,  efforts  are  underway  to  devise 
better  means  of  evaluating  these  types  of  benefits. 

Examples  of  extended  effects  include  ridership  and 
revenue  losses  experienced  by  transportation  providers  other 
than  the  one (s)  directly  involved  in  the  demonstration,  reduced 
transportation  costs  for  social  service  agencies  formerly 
supplying  transportation  for  their  clients,  and  economic  or 
environmental  changes  affecting  the  community  (e.g. , increase 
in  retail  sales) . In  addition,  it  is  sometimes  relevant  to  view 
the  benefits  of  a user's  increased  independence  from  the 
standpoint  of  the  person  who  formerly  provided  the 
transportation.  For  instance,  instituting  an  annual  pass  for 
youth  may  free  housewives  from  the  task  of  chauffeuring 
children  and  allow  them  opportunities  to  pursue  part-time 
employment . 
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5.8  Summary 


Once  user,  operator,  and  non-travel  impacts  have  been 
analyzed  in  detail,  there  is  a summary  section  of  the 
evaluation  report  which  deals  with  the  overall  feasibility 
and  impacts  of  the  demonstration  service  concept  or 
technique.  This  generally  includes  an  assessment  of  the 
degree  to  which  demonstration  objectives  have  been  achieved 
(in  particular,  the  applicable  SMD  objectives)  and  a 
discussion  of  the  major  evaluation  issues  which  synthesize 
findings  from  the  preceding  sections.  It  is  important  to 
note  that  in  this  final  stage  of  the  evaluation  process, 
both  the  degree  to  which  objectives  were  satisfied  and  the 
reasons  for  these  outcomes  are  essential  in  examining  the 
potential  of  the  service  concept  and  the  transferability  of 
findings. 
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